---
title: 'Online Appendix D'
header-includes:
   - \usepackage{pdfpages}
output: pdf_document
toc: true
---

```{r setup, include=FALSE}
require(knitr)
wd <- '~/Dropbox/Apps/ShareLaTeX/Minimal Effects/replication/data'
opts_knit$set(root.dir = wd)
```

```{r, echo=FALSE, message=FALSE}
master.sheet <- read.csv("master_sheet_output.csv", stringsAsFactors = FALSE)
source("../SI/representativeness_function.R")
source("../SI/balance_function.R")
source("../SI/balance_functionGOTV.R")
source("../SI/representativeness_functionGOTV.R")
source("../SI/results_table_function.R")
source("../SI/count_function.R")
```

**We apologize for the length of this Appendix section. Unfortunately given the number of experiments we discuss and our desire to be fully transparent, this is unavoidable. The key details of the experiments necessary for interpretation should all appear in the main text.**

# Overview

In this appendix, we describe the seven original field experiments, two difference-in-differences quasi-experiments, and two GOTV experiments that we conducted during the 2015 and 2016 election cycles.

All of these experiments were conducted with the same partner organization, Working America, the community affiliate of the AFL-CIO. Working America uses paid canvassers to go door-to-door persuading voters to support their endorsed candidates (typically Democrats) and encouraging voter turnout. 

Each of these experiments followed a standard model, using the online panel plus placebo procedure described in Broockman, Kalla, and Sekhon (2017): 

1. Working America would define an experimental universe of voters they believed to be persuadable. 
2. A polling division would then send these voters a letter encouraging them to participate in a paid, online survey. This survey would include multiple questions on political, social, and local issues. Neither the survey nor the letter would mention Working America. As part of the survey, the polling division would then collect the voters' cell phone numbers and email addresses.
3. Among the voters who completed the survey and provided their contact information, Working America would randomly assign half to a treatment group that would be canvassed with Working America's typical persuasion message and half to a placebo group that would receive an unrelated canvass, typically on ascertaining sources of news consumption. The placebo contained no persuasion messaging and was only used to identify compliers, those voters who, had they been in treatment, would have opened their doors.
4. Working America would send the polling division the list of compliers. The polling division would then resurvey the compliers several days after the initial canvass with a similar survey on political, social, and local issues.
5. Working America would then send the authors the survey data to conduct analyses of their canvassing.

In the experimental analyses, we followed two standard procedures from Broockman, Kalla, and Sekhon (2017):

1. The surveys typically included multiple questions on the race that was the subject of the persuasion effort. Typically, these questions were a horse-race and a candidate favorability question for both the Democrat and Republican. When multiple questions were available, we would combine them into a single index designed to reduce measurement error. In all cases, we take the first dimension from the factor analysis output, then rescale this factor such that the placebo group has a mean 0 and standard deviation of 1. This allows us to interpret the treatment effects as the effect in standard deviations the treatment would have among an untreated population. The factor analysis and rescaling code came from the supplementary materials of Broockman and Kalla (2016).
2. Our main analysis for each experiment was always a regression of the factor (described above) on a treatment indicator and a set of pre-treatment covariates, with household-level cluster-robust standard errors. The pre-treatment covariates used were always the same as those used in Working America's balance tests before canvassing. The use of pre-treatment covariates that are highly predictive of the outcome noticeably decreases sampling variability and increases statistical power. 

# PA Experiment, 2015, Mayor Primary

This experiment was conducted during the 2015 Philadelphia mayoral Democratic primary. Working America canvassed to increase support for Jim Kenney. Canvassing took place from `r master.sheet[which(master.sheet$Data == "Philly_Canvass_2015_anon_immed.dta"),"Date.Canvass"]`. An initial post-treatment survey took place from `r master.sheet[which(master.sheet$Data == "Philly_Canvass_2015_anon_immed.dta"),"Date.t1"]`. A second follow-up post-treatment survey took place from `r master.sheet[which(master.sheet$Data == "Philly_Canvass_2015_anon_late.dta"),"Date.t1"]`. The election was held on 5/19/15. 

## Experimental Universe

Below, we describe the representativeness of the experimental universe. This first table compares the responders to the initial post-treatment survey to everyone who was canvassed.

```{r, echo=FALSE}
representativeness.function("Philly_Canvass_2015_anon_immed.dta")
```

This second table compares the responders to the second follow-up post-treatment survey to everyone who was canvassed.

```{r, echo=FALSE}
representativeness.function("Philly_Canvass_2015_anon_late.dta")
```

## Tests of Covariate Balance and Differential Attrition

Below, we report covariate balance across treatment and placebo at each of three stages: at the time of canvassing, at the time of the initial post-treatment survey, and at the time of the follow-up post-treatment survey. We do this by regressing a treatment indicator on all of the covariates. Each p-value reports whether that covariate is predictive of treatment assignment. In expectation, from random assignment, the covariates should be independent of treatment assignment. As a summary statistics, we also report the F-statistic from this multivariate regression. 

This table shows covariate balance among everyone canvassed.
```{r, echo=FALSE}
balance.function("Philly_Canvass_2015_anon_immed.dta")
```

This table shows covariate balance among everyone who took the initial post-treatment survey.
```{r, echo=FALSE}
balance.function("Philly_Canvass_2015_anon_immed.dta", "data$respondent_t1 == 1")
```

This table shows covariate balance among everyone who took the follow-up post-treatment survey.
```{r, echo=FALSE}
balance.function("Philly_Canvass_2015_anon_late.dta", "data$respondent_t1 == 1")
```

We also present the number of individuals, by treatment condition, at each stage.

The first table is for the immediate post-treatment survey.
```{r, echo=FALSE}
count.function("Philly_Canvass_2015_anon_immed.dta")
```

This second table is for the follow-up post-treatment survey.
```{r, echo=FALSE}
count.function("Philly_Canvass_2015_anon_late.dta")
```


## Description of Treatment

\includepdf[pages={1-},scale=0.75]{"scripts/PA Experiment_2015_Mayor_Primary"}

## Outcome Measures

1. In the upcoming Democratic Primary election to nominate a candidate for Mayor of Philadelphia, which of the following candidates would you vote for?
2. Kenney Favorability.

## Results

This first table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("Philly_Canvass_2015_anon_immed.dta", "Mayor")
```

This second table shows the experimental results of the canvass, as measured in the follow-up post-treatment survey.
```{r, echo=FALSE}
results.table("Philly_Canvass_2015_anon_late.dta", "Mayor")
```

# WA Experiment, 2015, State Legislator

This experiment was conducted during the 2015 Washington state special election in State House District 30b. Working America canvassed to increase support for Carol Gregory. Gregory was appointed to fill the seat after Roger Freeman passed away. The special election was held to determine who would hold the seat for the remainder of Freeman's term.

Canvassing took place from `r master.sheet[which(master.sheet$Data == "Wash_Canvass_2015_anon.dta"),"Date.Canvass"]`. An initial post-treatment survey took place from `r master.sheet[which(master.sheet$Data == "Wash_Canvass_2015_anon.dta"),"Date.t1"]`. A second follow-up post-treatment survey took place from `r master.sheet[which(master.sheet$Data == "Wash_Canvass_2015_anont2.dta"),"Date.t1"]`. The election was held on 11/3/15. 

## Experimental Universe

Below, we describe the representativeness of the experimental universe. This first table compares the responders to the initial post-treatment survey to everyone who was canvassed.

```{r, echo=FALSE}
representativeness.function("Wash_Canvass_2015_anon.dta")
```

This second table compares the responders to the second follow-up post-treatment survey to everyone who was canvassed.

```{r, echo=FALSE}
representativeness.function("Wash_Canvass_2015_anont2.dta")
```

## Tests of Covariate Balance and Differential Attrition

Below, we report covariate balance across treatment and placebo at each of three stages: at the time of canvassing, at the time of the initial post-treatment survey, and at the time of the follow-up post-treatment survey. We do this by regressing a treatment indicator on all of the covariates. Each p-value reports whether that covariate is predictive of treatment assignment. In expectation, from random assignment, the covariates should be independent of treatment assignment. As a summary statistics, we also report the F-statistic from this multivariate regression. 

This table shows covariate balance among everyone canvassed.
```{r, echo=FALSE}
balance.function("Wash_Canvass_2015_anon.dta")
```

This table shows covariate balance among everyone who took the initial post-treatment survey.
```{r, echo=FALSE}
balance.function("Wash_Canvass_2015_anon.dta", "data$respondent_t1 == 1")
```

This table shows covariate balance among everyone who took the follow-up post-treatment survey.
```{r, echo=FALSE}
balance.function("Wash_Canvass_2015_anont2.dta", "data$respondent_t1 == 1")
```

We also present the number of individuals, by treatment condition, at each stage.

The first table is for the immediate post-treatment survey.
```{r, echo=FALSE}
count.function("Wash_Canvass_2015_anon.dta")
```

This second table is for the follow-up post-treatment survey.
```{r, echo=FALSE}
count.function("Wash_Canvass_2015_anont2.dta")
```

## Description of Treatment and Placebo

\includepdf[pages={1-},scale=0.75]{"scripts/WA Experiment_2015_StateLeg"}

## Outcome Measures

1. Vote choice.
2. Gregory favorability.
3. Hickel favorability.
4. Which candidate do you think would do a better job representing people like you?.

## Results

This first table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("Wash_Canvass_2015_anon.dta", "State Legislator")
```

This second table shows the experimental results of the canvass, as measured in the follow-up post-treatment survey.
```{r, echo=FALSE}
results.table("Wash_Canvass_2015_anont2.dta", "State Legislator")
```

# OH Experiment 1, 2016, Senate

This experiment was conducted early in Ohio's Senate election. Working America canvassed to increase support for Ted Strickland. At this point, Working America had not yet begun working on the presidential race. Canvassing took place from `r master.sheet[which(master.sheet$Data == "OH_EarlyCanvass_2016.dta"),"Date.Canvass"]`. An initial post-treatment survey took place from `r master.sheet[which(master.sheet$Data == "OH_EarlyCanvass_2016.dta"),"Date.t1"]`. The election was held on 11/8/16. 

## Experimental Universe

Below, we describe the representativeness of the experimental universe. This table compares the responders to the initial post-treatment survey to everyone who was canvassed.

```{r, echo=FALSE}
representativeness.function("OH_EarlyCanvass_2016.dta")
```

## Tests of Covariate Balance and Differential Attrition

Below, we report covariate balance across treatment and placebo at each of two stages: at the time of canvassing and at the time of the initial post-treatment survey. We do this by regressing a treatment indicator on all of the covariates. Each p-value reports whether that covariate is predictive of treatment assignment. In expectation, from random assignment, the covariates should be independent of treatment assignment. As a summary statistics, we also report the F-statistic from this multivariate regression. 

This table shows covariate balance among everyone canvassed.
```{r, echo=FALSE}
balance.function("OH_EarlyCanvass_2016.dta")
```

This table shows covariate balance among everyone who took the initial post-treatment survey.
```{r, echo=FALSE}
balance.function("OH_EarlyCanvass_2016.dta", "data$respondent_t1 == 1")
```

We also present the number of individuals, by treatment condition, at each stage.

```{r, echo=FALSE}
count.function("OH_EarlyCanvass_2016.dta")
```

## Description of Treatment

Note that two different treatment scripts were used. Because there was no statistically significant differences between the efficacy of the two scripts, we merged them for the purposes of our analysis.

\includepdf[pages={1-},scale=0.75]{"scripts/OH Experiment_2016_Num1_Senate"}

## Outcome Measures

1. Do you approve or disapprove of the way Rob Portman is handling his job as senator?
2. Do you have a favorable or unfavorable opinion of Ted Strickland?
3. Ohio also has a Senate election this fall between current Senator Republican Rob Portman and Democrat Ted Strickland. How do you plan on voting?
4. When it comes to representing Ohio in the U.S. Senate, which candidate do you think is best qualified, Democrat Ted Strickland or Republican Rob Portman?

## Results

This table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("OH_EarlyCanvass_2016.dta", "Senate")
```

# OH Experiment 2, 2016, President and Senate

This experiment was conducted later in Ohio's Senate election, and also included persuasion on the presidential race. This was a distinct experimental universe from the first test. Working America canvassed to increase support for Ted Strickland and Hillary Clinton. Canvassing took place from `r master.sheet[which(master.sheet$Data == "OH_LateCanvass_2016.dta" & master.sheet$Seat == "President"),"Date.Canvass"]`. An initial post-treatment survey took place from `r master.sheet[which(master.sheet$Data == "OH_LateCanvass_2016.dta" & master.sheet$Seat == "President"),"Date.t1"]`. A second follow-up post-treatment survey took place from `r master.sheet[which(master.sheet$Data == "OH_LateCanvass_2016_ED.dta" & master.sheet$Seat == "President"),"Date.t1"]`. The election was held on 11/8/16. 

## Experimental Universe

Below, we describe the representativeness of the experimental universe. This table compares the responders to the initial post-treatment survey to everyone who was canvassed.

```{r, echo=FALSE}
representativeness.function("OH_LateCanvass_2016.dta")
```

This second table compares the responders to the second follow-up post-treatment survey to everyone who was canvassed.

```{r, echo=FALSE}
representativeness.function("OH_LateCanvass_2016_ED.dta")
```

## Tests of Covariate Balance and Differential Attrition

Below, we report covariate balance across treatment and placebo at each of three stages: at the time of canvassing, at the time of the initial post-treatment survey, and at the time of the follow-up post-treatment survey. We do this by regressing a treatment indicator on all of the covariates. Each p-value reports whether that covariate is predictive of treatment assignment. In expectation, from random assignment, the covariates should be independent of treatment assignment. As a summary statistics, we also report the F-statistic from this multivariate regression. 

This table shows covariate balance among everyone canvassed.
```{r, echo=FALSE}
balance.function("OH_LateCanvass_2016.dta")
```

This table shows covariate balance among everyone who took the initial post-treatment survey.
```{r, echo=FALSE}
balance.function("OH_LateCanvass_2016.dta", "data$respondent_t1 == 1")
```

This table shows covariate balance among everyone who took the follow-up post-treatment survey.
```{r, echo=FALSE}
balance.function("OH_LateCanvass_2016_ED.dta", "data$respondent_t1 == 1")
```

We also present the number of individuals, by treatment condition, at each stage.

The first table is for the immediate post-treatment survey.
```{r, echo=FALSE}
count.function("OH_LateCanvass_2016.dta")
```

This second table is for the follow-up post-treatment survey.
```{r, echo=FALSE}
count.function("OH_LateCanvass_2016_ED.dta")
```

## Description of Treatment

\includepdf[pages={1-},scale=0.75]{"scripts/OH Experiment_2016_Num2_POTUSSenate"}

## Outcome Measures

President:

1. Thinking about the current presidential election, if the presidential election were being held today between Democrat Hillary Clinton, Republican Donald Trump, Libertarian Gary Johnson and Green Party candidate Jill Stein, who would you vote for?
2. Do you have a favorable or unfavorable opinion of Hillary Clinton?
3. Do you have a favorable or unfavorable opinion of Donald Trump?
4. When it comes to being President, which candidate is best qualified, Republican Donald Trump or Democrat Hillary Clinton?

Senate:
1. Do you approve or disapprove of the way Rob Portman is handling his job as senator?
2. Do you have a favorable or unfavorable opinion of Ted Strickland?
3. Ohio also has a Senate election this fall between current Senator Republican Rob Portman and Democrat Ted Strickland. How do you plan on voting?
4. When it comes to representing Ohio in the U.S. Senate, which candidate do you think is best qualified, Democrat Ted Strickland or Republican Rob Portman?

## Results

### President

This first table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("OH_LateCanvass_2016.dta", "President")
```

This second table shows the experimental results of the canvass, as measured in the follow-up post-treatment survey.
```{r, echo=FALSE}
results.table("OH_LateCanvass_2016_ED.dta", "President")
```

### Senate

This first table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("OH_LateCanvass_2016.dta", "Senate")
```

This second table shows the experimental results of the canvass, as measured in the follow-up post-treatment survey.
```{r, echo=FALSE}
results.table("OH_LateCanvass_2016_ED.dta", "Senate")
```

# NC Experiment, 2016, President, Senate, Governor, Supreme Court

This experiment was conducted during the 2016 North Carolina general election. Working America canvassed to increase support for Hillary Clinton and Deborah Ross. As part of these canvasses, North Carolina also distributed literature to increase support for Roy Cooper and Michael Morgan, a Supreme Court candidate. Canvassing took place from `r master.sheet[which(master.sheet$Data == "NC_Canvass_2016.dta" & master.sheet$Seat == "President"),"Date.Canvass"]`. An initial post-treatment survey took place from `r master.sheet[which(master.sheet$Data == "NC_Canvass_2016.dta" & master.sheet$Seat == "President"),"Date.t1"]`. The election was held on 11/8/16. 

## Experimental Universe

Below, we describe the representativeness of the experimental universe. This table compares the responders to the initial post-treatment survey to everyone who was canvassed.

```{r, echo=FALSE}
representativeness.function("NC_Canvass_2016.dta")
```

## Tests of Covariate Balance and Differential Attrition

Below, we report covariate balance across treatment and placebo at each of two stages: at the time of canvassing and at the time of the initial post-treatment survey. We do this by regressing a treatment indicator on all of the covariates. Each p-value reports whether that covariate is predictive of treatment assignment. In expectation, from random assignment, the covariates should be independent of treatment assignment. As a summary statistics, we also report the F-statistic from this multivariate regression. 

This table shows covariate balance among everyone canvassed.
```{r, echo=FALSE}
balance.function("NC_Canvass_2016.dta")
```

This table shows covariate balance among everyone who took the initial post-treatment survey.
```{r, echo=FALSE}
balance.function("NC_Canvass_2016.dta", "data$respondent_t1 == 1")
```

We also present the number of individuals, by treatment condition, at each stage.

```{r, echo=FALSE}
count.function("NC_Canvass_2016.dta")
```

## Description of Treatment

Below we include the script that was used, as well as the literature that was given at the door on the Supreme Court and gubernatorial races. These were not explicitly mentioned in the script, which focused on the presidential and senate races. 

\includepdf[pages={1-},scale=0.75]{"scripts/NC Experiment and Diff_2016"}
\includepdf[pages={1-},scale=0.75]{"scripts/NC Experiment and Diff Court Lit"}
\includepdf[pages={1-},scale=0.75]{"scripts/NC Experiment and Diff Gov Lit"}


## Outcome Measures

President:

1. If the election for president were held today between Democrat Hillary Clinton and Republican Donald Trump, who would you vote for?
2. Do you have a favorable or unfavorable opinion of Hillary Clinton?
3. Do you have a favorable or unfavorable opinion of Donald Trump?
4. When it comes to being president, which candidate is best qualified, Republican Donald Trump or Democrat Hillary Clinton?

Senate (only one question was asked):

1. North Carolina also has a Senate election this fall. If the election were held today between current senator Republican Richard Burr and Democrat former State Representative Deborah Ross, how do you think you would vote?

Governor:

1. North Carolina will also hold elections for governor this fall. If the election were held today between current Governor Republican Pat McCrory and Democratic Attorney General Roy Cooper, how do you think you would vote?
2. Do you approve or disapprove of the way Pat McCrory is handling his job as governor?
3. Do you approve or disapprove of the way Roy Cooper is handling his job as attorney general?

Supreme Court:

1. Do you have a favorable or unfavorable opinion of Supreme Court Justice Robert Edmunds?
2. Do you have a favorable or unfavorable opinion of Wake Country Judge Michael Morgan?
3. If the election for North Carolina Supreme Court justice were held tomorrow between Robert Edmunds and Michael Morgan, who would you vote for?

## Results

### President

This first table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("NC_Canvass_2016.dta", "President")
```

### Senate

This first table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("NC_Canvass_2016.dta", "Senate")
```

### Governor

This first table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("NC_Canvass_2016.dta", "Governor")
```

### Supreme Court

This first table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("NC_Canvass_2016.dta", "Supreme Court")
```

# FL Experiment, 2016, Generic Democratic Candidates

This experiment was conducted during FL's 2016 general election. Working America canvassed to increase support for Hillary Clinton and Democratic candidates more generally. Canvassing took place from `r master.sheet[which(master.sheet$Data == "FL_Canvass_2016.dta" & master.sheet$Seat == "President"),"Date.Canvass"]`. An initial post-treatment survey took place from `r master.sheet[which(master.sheet$Data == "FL_Canvass_2016.dta" & master.sheet$Seat == "President"),"Date.t1"]`. The election was held on 11/8/16. 

## Experimental Universe

Below, we describe the representativeness of the experimental universe. This table compares the responders to the initial post-treatment survey to everyone who was canvassed.

```{r, echo=FALSE}
representativeness.function("FL_Canvass_2016.dta")
```

## Tests of Covariate Balance and Differential Attrition

Below, we report covariate balance across treatment and placebo at each of two stages: at the time of canvassing and at the time of the initial post-treatment survey. We do this by regressing a treatment indicator on all of the covariates. Each p-value reports whether that covariate is predictive of treatment assignment. In expectation, from random assignment, the covariates should be independent of treatment assignment. As a summary statistics, we also report the F-statistic from this multivariate regression. 

This table shows covariate balance among everyone canvassed.
```{r, echo=FALSE}
balance.function("FL_Canvass_2016.dta")
```

This table shows covariate balance among everyone who took the initial post-treatment survey.
```{r, echo=FALSE}
balance.function("FL_Canvass_2016.dta", "data$respondent_t1 == 1")
```

We also present the number of individuals, by treatment condition, at each stage.

```{r, echo=FALSE}
count.function("FL_Canvass_2016.dta")
```

## Description of Treatment

This experiment attempted to persuade voters to vote for Democratic candidates in general, not any one particular candidate. As a result, we use an index of voters' votes across multiple races as the outcome.

\includepdf[pages={1-},scale=0.75]{"scripts/FL Experiment_2016_President"}

## Outcome Measures

1. If the election for President were held today between Democrat Hillary Clinton and Republican Donald Trump, who would you vote for?
2. Trump Feeling Thermometer.
3. Clinton Feeling Thermometer.
4. Florida also has a Senate election this fall. If the election were held today between current senator Republican Marco Rubio and Democrat Representative Patrick Murphy, how do you think you would vote?
5. Florida will have an election for governor coming up in a few years. If the election were held today, do you think you would vote for the Democratic candidate or Republican candidate?
6. Your area will also have an election for US Congress this year. If the election were held today between Democrat Darren Soto and Republican Wayne Liebnitzky, how do you think you would vote?
7. Now, thinking about Florida's state senate, if the election for state senator were held today between Republican Dean Asher and Democrat Linda Stewart, how do you think you would vote?

## Results

This first table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("FL_Canvass_2016.dta", "Dem Candidates")
```

# MO Experiment, 2016, Governor

This experiment was conducted during Missouri's gubernatorial race. Working America canvassed to increase support for Chris Koster. Canvassing took place from `r master.sheet[which(master.sheet$Data == "MO_Canvass_2016.dta" & master.sheet$Seat == "Governor"),"Date.Canvass"]`. An initial post-treatment survey took place from `r master.sheet[which(master.sheet$Data == "MO_Canvass_2016.dta" & master.sheet$Seat == "Governor"),"Date.t1"]`. The election was held on 11/8/16. 

## Experimental Universe

Below, we describe the representativeness of the experimental universe. This table compares the responders to the initial post-treatment survey to everyone who was canvassed.

```{r, echo=FALSE}
representativeness.function("MO_Canvass_2016.dta")
```

## Tests of Covariate Balance and Differential Attrition

Below, we report covariate balance across treatment and placebo at each of two stages: at the time of canvassing and at the time of the initial post-treatment survey. We do this by regressing a treatment indicator on all of the covariates. Each p-value reports whether that covariate is predictive of treatment assignment. In expectation, from random assignment, the covariates should be independent of treatment assignment. As a summary statistics, we also report the F-statistic from this multivariate regression. 

This table shows covariate balance among everyone canvassed.
```{r, echo=FALSE}
balance.function("MO_Canvass_2016.dta")
```

This table shows covariate balance among everyone who took the initial post-treatment survey.
```{r, echo=FALSE}
balance.function("MO_Canvass_2016.dta", "data$respondent_t1 == 1")
```

## Description of Treatment

\includepdf[pages={1-},scale=0.75]{"scripts/MO Experiment_2016_Governor"}

## Outcome Measures

1. Missouri will also hold elections for governor this fall. If the election were held today between Republican Eric Greitens and Democrat Chris Koster, how do you think you would vote?
2. Do you have a favorable or unfavorable opinion of Eric Greitens?
3. Do you have a favorable or unfavorable opinion of Chris Koster?

## Results

This first table shows the experimental results of the canvass, as measured in the initial post-treatment survey. We present results both controlling for the pre-treatment covariates used in the test of covariate balance and without.
```{r, echo=FALSE}
results.table("MO_Canvass_2016.dta", "Governor")
```

# NC GOTV Experiment, 2016

Using a distinct experimental universe but the same canvassers, Working America conducted a voter turnout experiment in the 2016 general election in North Carolina. Canvassing took place from 24 October 2016 through Election Day, 8 November 2016. 

## Experimental Universe

```{r, include=FALSE, message=FALSE}
nc.gotv <- read.dta('NC_GOTV2016_anon.dta')
```

The experiment consisted of `r nrow(nc.gotv)` people randomly assigned to one of three treatment conditions: a GOTV canvass, a placebo canvass, and a pure control group. 

Randomization was conducted based on the number of registered voters in a precinct. In precincts with over 1,000 registered voters, approximately 10% of households were randomly assigned to control, 5% to placebo, and 85% to treatment. In precincts with less than 1,000 registered voters, approximately 5% of households were randomly assigned to placebo and 95% to treatment. 

```{r, echo=FALSE}
representativeness.function.gotv(nc.gotv, c("general15", "general14", "general13", 
                                       "general12", "general11", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "vf_dem", 
                                       "vf_rep", "vf_female", "precinct_1k"))
```

## Tests of Covariate Balance

Below we present the covariate balance at each stage (assignment, attempted, and canvassed) and by precinct type (more or less than 1,000 voters).

First, we present balance at the **assignment** stage among voters living in precincts with **more** than 1,000 voters.
```{r, echo=FALSE}
balance.function.gotv(experiment = nc.gotv, vars = c("general15", "general14", "general13", 
                                       "general12", "general11", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "vf_dem", 
                                       "vf_rep", "vf_female", "precinct_1k"),
                      define.subset = "experiment$precinct_1k == 1")
```

Next, we present balance at the **attempted** stage among voters living in precincts with **more** than 1,000 voters.
```{r, echo=FALSE}
balance.function.gotv(experiment = nc.gotv, vars = c("general15", "general14", "general13", 
                                       "general12", "general11", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "vf_dem", 
                                       "vf_rep", "vf_female", "precinct_1k"),
                      define.subset = "experiment$precinct_1k == 1 & experiment$attempted == 1")
```

Finally, we present balance at the **canvassed** stage among voters living in precincts with **more** than 1,000 voters.
```{r, echo=FALSE}
balance.function.gotv(experiment = nc.gotv, vars = c("general15", "general14", "general13", 
                                       "general12", "general11", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "vf_dem", 
                                       "vf_rep", "vf_female", "precinct_1k"),
                      define.subset = "experiment$precinct_1k == 1 & experiment$canvassed == 1")
```


Second, we present balance at the **assignment** stage among voters living in precincts with **less** than 1,000 voters.
```{r, echo=FALSE}
balance.function.gotv(experiment = nc.gotv, vars = c("general15", "general14", "general13", 
                                       "general12", "general11", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "vf_dem", 
                                       "vf_rep", "vf_female", "precinct_1k"),
                      define.subset = "experiment$precinct_1k == 0")
```

Next, we present balance at the **attempted** stage among voters living in precincts with **less** than 1,000 voters.
```{r, echo=FALSE}
balance.function.gotv(experiment = nc.gotv, vars = c("general15", "general14", "general13", 
                                       "general12", "general11", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "vf_dem", 
                                       "vf_rep", "vf_female", "precinct_1k"),
                      define.subset = "experiment$precinct_1k == 0 & experiment$attempted == 1")
```

Finally, we present balance at the **canvassed** stage among voters living in precincts with **less** than 1,000 voters.
```{r, echo=FALSE}
balance.function.gotv(experiment = nc.gotv, vars = c("general15", "general14", "general13", 
                                       "general12", "general11", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "vf_dem", 
                                       "vf_rep", "vf_female", "precinct_1k"),
                      define.subset = "experiment$precinct_1k == 0 & experiment$canvassed == 1")
```

## Description of Treatment

Below is the GOTV script used in North Carolina. See Question 4 for the voter turnout component.

\includepdf[pages={1-},scale=0.75]{"scripts/NC GOTV"}

## Results

```{r, echo=FALSE}
se <- function(x) sqrt(var(x)/length(x))
row.results <- function(experiment, condition) {
  overall.mean <- mean(experiment[which(eval(parse(text = condition))),"voted16"])
  overall.se <- se(experiment[which(eval(parse(text = condition))),"voted16"])
  attempted.mean <- mean(experiment[which(eval(parse(text = condition)) 
                                       & experiment$attempted == 1),"voted16"])
  attempted.se <- se(experiment[which(eval(parse(text = condition)) 
                                   & experiment$attempted == 1),"voted16"])
  canvassed.mean <- mean(experiment[which(eval(parse(text = condition))
                                       & experiment$canvassed == 1),"voted16"])
  canvassed.se <- se(experiment[which(eval(parse(text = condition)) 
                                   & experiment$canvassed == 1),"voted16"])
  return(list(overall.mean = round(overall.mean, digits = 3),
              overall.se = round(overall.se, digits = 3),
              attempted.mean = round(attempted.mean, digits = 3),
              attempted.se = round(attempted.se, digits = 3),
              canvassed.mean = round(canvassed.mean, digits = 3),
              canvassed.se = round(canvassed.se, digits = 3)))
}

control.more.1k <- row.results(nc.gotv, "experiment$treat == -1 & experiment$precinct_1k == 1")
placebo.more.1k <- row.results(nc.gotv, "experiment$treat == 0 & experiment$precinct_1k == 1")
treat.more.1k <- row.results(nc.gotv, "experiment$treat == 1 & experiment$precinct_1k == 1")

control.less.1k <- row.results(nc.gotv, "experiment$treat == -1 & experiment$precinct_1k == 0")
placebo.less.1k <- row.results(nc.gotv, "experiment$treat == 0 & experiment$precinct_1k == 0")
treat.less.1k <- row.results(nc.gotv, "experiment$treat == 1 & experiment$precinct_1k == 0")
```

Condition | Overall | Attempted | Canvassed
--------| ----------| ----------| ----------
Control, >1k Precinct | `r control.more.1k$overall.mean` (`r control.more.1k$overall.se`) | `r control.more.1k$attempted.mean` (`r control.more.1k$attempted.se`) | `r control.more.1k$canvassed.mean` (`r control.more.1k$canvassed.se`)
Placebo, >1k Precinct | `r placebo.more.1k$overall.mean` (`r placebo.more.1k$overall.se`) | `r placebo.more.1k$attempted.mean` (`r placebo.more.1k$attempted.se`) | `r placebo.more.1k$canvassed.mean` (`r placebo.more.1k$canvassed.se`)
Treatment, >1k Precinct | `r treat.more.1k$overall.mean` (`r treat.more.1k$overall.se`) | `r treat.more.1k$attempted.mean` (`r treat.more.1k$attempted.se`) | `r treat.more.1k$canvassed.mean` (`r treat.more.1k$canvassed.se`)
--------| ----------| ----------| ----------
Control, <1k Precinct | `r control.less.1k$overall.mean` (`r control.less.1k$overall.se`) | `r control.less.1k$attempted.mean` (`r control.less.1k$attempted.se`) | `r control.less.1k$canvassed.mean` (`r control.less.1k$canvassed.se`)
Placebo, <1k Precinct | `r placebo.less.1k$overall.mean` (`r placebo.less.1k$overall.se`) | `r placebo.less.1k$attempted.mean` (`r placebo.less.1k$attempted.se`) | `r placebo.less.1k$canvassed.mean` (`r placebo.less.1k$canvassed.se`)
Treatment, <1k Precinct | `r treat.less.1k$overall.mean` (`r treat.less.1k$overall.se`) | `r treat.less.1k$attempted.mean` (`r treat.less.1k$attempted.se`) | `r treat.less.1k$canvassed.mean` (`r treat.less.1k$canvassed.se`)

Note: Each cell denotes the turnout rate (mean and standard error of the mean) for each condition and by precinct type at each stage in the experiment.

To estimate a complier average causal effect (CACE) pooled across the two types of precincts, we compare the turnout rates among just those voters canvassed in the treatment and placebo conditions. We do this by regressing turnout on an indicator for treatment and an indicator for precinct type (more or less than 1,000 voters). In one model, we also include covariates from the 2015, 2014, 2012, 2010, 2008, and 2006 general elections and the 2016 primary election. As stated in Version 1.05 of *Standard operating procedures for Don Green’s lab at Columbia*, "If the PAP fails to specify the choice of covariates for regression adjustment or for the test of covariate balance, the default set of covariates will include voter turnout in all past elections for which data are available in the voter file, excluding any elections in which turnout rates in the subject pool were below 5%." [http://htmlpreview.github.io/?https://github.com/acoppock/Green-Lab-SOP/blob/master/Green_Lab_SOP.html](http://htmlpreview.github.io/?https://github.com/acoppock/Green-Lab-SOP/blob/master/Green_Lab_SOP.html). Furthermore, all standard errors are cluster-robust at the household level, which was the unit of treatment assignment.

```{r, include=FALSE, message=FALSE}
library(sandwich)
library(lmtest)

cl   <- function(fm, cluster){
  M <- length(unique(cluster))
  N <- length(cluster)
  K <- fm$rank
  dfc <- (M/(M-1))*((N-1)/(N-K))
  uj  <- apply(estfun(fm), 2, function(x) tapply(x, cluster, sum))
  vcovCL <- dfc*sandwich(fm, meat=crossprod(uj)/N)
  coeftest(fm, vcovCL)
}

nc.gotv <- subset(nc.gotv, canvassed == 1)
vars <- c("general15", "general14", "general12", "general10", "general08",
          "general06", "primary16")
x <- as.matrix(nc.gotv[,vars])

nc.no.covars.gotv <- round(cl(lm(nc.gotv$voted16 ~ nc.gotv$treat + nc.gotv$precinct_1k), nc.gotv$hh_id)[2,], digits = 4) 
nc.no.covars.gotv[c("Estimate", "Std. Error")] <- nc.no.covars.gotv[c("Estimate", "Std. Error")] * 100 #Get in PP

nc.covars.gotv <- round(cl(lm(nc.gotv$voted16 ~ nc.gotv$treat + nc.gotv$precinct_1k + x), nc.gotv$hh_id)[2,], digits = 4) 
nc.covars.gotv[c("Estimate", "Std. Error")] <- nc.covars.gotv[c("Estimate", "Std. Error")] * 100 #Get in PP
```

Without covariates, we estimate a treatment effect of `r nc.no.covars.gotv["Estimate"]` (SE = `r nc.no.covars.gotv["Std. Error"]`). With covariates, we estimate a treatment effect of `r nc.covars.gotv["Estimate"]` (SE = `r nc.covars.gotv["Std. Error"]`).

This allows us to conclude that Working America's GOTV canvass increased turnout with a CACE of approximately 2 percentage points. To contextualize this, Table A-2 of Green and Gerber (2015) presents a meta-analysis of the CACE effects for door-to-door GOTV canvassing by base rate of turnout in the control group. Their meta-analysis suggests that the average CACE in a race when the turnout rate in the control group is between 50-70% is 1.4 percentage points (in this NC experiment, it was 68% among compliers in the placebo group). Thus, the Working America GOTV effect of 2 percentage points is apprximately 43% more effective than the average effect. 

# MO GOTV Experiment, 2016

Using a distinct experimental universe but the same canvassers, Working America conducted a voter turnout experiment in the 2016 general election in Missouri. Canvassing took place from 25 October 2016 through Election Day, 8 November 2016.

As we discuss more below, this experiment suffered from an implementation error which led to covariate imbalance between the compliers in the treatment and placebo groups. We therefore excluded this experiment from the main text. 

## Experimental Universe

```{r, include=FALSE, message=FALSE}
mo.gotv <- read.dta('MO_GOTV2016_anon.dta')
```

The experiment consisted of `r nrow(mo.gotv)` people randomly assigned to one of three treatment conditions: a GOTV canvass, a placebo canvass, and a pure control group. 

Randomization was conducted by city. In the city of St. Louis, approximately 20% of households were randomly assigned to control, 5% to placebo, and 75% to treatment. In the county of St. Louis, approximately 40% of households were randomly assigned to control, 5% of households were randomly assigned to placebo and 55% to treatment. 

Note that we do not have party registration data for Missouri.

```{r, echo=FALSE}
representativeness.function.gotv(mo.gotv, c("general14", "general13", 
                                       "general12", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "st_louis_city"))
```

## Tests of Covariate Balance

Below we present the covariate balance at each stage (assignment, attempted, and canvassed) and by city/county of St. Louis. In particular, note the covariate imbalance at the canvassed stage among voters living in St Louis City. 

First, we present balance at the **assignment** stage among voters living in St Louis **City**.
```{r, echo=FALSE}
balance.function.gotv(experiment = mo.gotv, vars = c("general14", "general13", 
                                       "general12", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "st_louis_city"),
                      define.subset = "experiment$st_louis_city == 1")
```

Next, we present balance at the **attempted** stage among voters living in St Louis **City**.
```{r, echo=FALSE}
balance.function.gotv(experiment = mo.gotv, vars = c("general14", "general13", 
                                       "general12", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "st_louis_city"),
                      define.subset = "experiment$st_louis_city == 1 & experiment$attempted == 1")
```

Finally, we present balance at the **canvassed** stage among voters living in St Louis **City**. The differences in voter turnout between treatment and placebo on the 2014, 2012, and 2006 general elections are worrisome, suggesting some imbalance in treatment delivery. Ex ante, the compliers in the placebo group appear to be more likely to vote than the compliers in the treatment group. 

```{r, echo=FALSE}
balance.function.gotv(experiment = mo.gotv, vars = c("general14", "general13", 
                                       "general12", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "st_louis_city"),
                      define.subset = "experiment$st_louis_city == 1 & experiment$canvassed == 1")
```

Second, we present balance at the **assignment** stage among voters living in St Louis **County**. Note that a small number of control subjects were accidentally attempted. 
```{r, echo=FALSE}
balance.function.gotv(experiment = mo.gotv, vars = c("general14", "general13", 
                                       "general12", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "st_louis_city"),
                      define.subset = "experiment$st_louis_city == 0")
```

Next, we present balance at the **attempted** stage among voters living in St Louis **County**. Note that a small number of control subjects were accidentally canvassed.
```{r, echo=FALSE}
balance.function.gotv(experiment = mo.gotv, vars = c("general14", "general13", 
                                       "general12", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "st_louis_city"),
                      define.subset = "experiment$st_louis_city == 0 & experiment$attempted == 1")
```

Finally, we present balance at the **canvassed** stage among voters living in St Louis **County**.
```{r, echo=FALSE}
balance.function.gotv(experiment = mo.gotv, vars = c("general14", "general13", 
                                       "general12", "general10", 
                                       "general09", "general08", "general07", 
                                       "general06", "primary16", "st_louis_city"),
                      define.subset = "experiment$st_louis_city == 0 & experiment$canvassed == 1")
```

## Description of Treatment

The Missouri GOTV script followed a similar outline as the North Carolina one. See above for more details.

## Results

```{r, echo=FALSE}
control.more.1k <- row.results(mo.gotv, "experiment$treat == -1 & experiment$st_louis_city == 1")
placebo.more.1k <- row.results(mo.gotv, "experiment$treat == 0 & experiment$st_louis_city == 1")
treat.more.1k <- row.results(mo.gotv, "experiment$treat == 1 & experiment$st_louis_city == 1")

control.less.1k <- row.results(mo.gotv, "experiment$treat == -1 & experiment$st_louis_city == 0")
placebo.less.1k <- row.results(mo.gotv, "experiment$treat == 0 & experiment$st_louis_city == 0")
treat.less.1k <- row.results(mo.gotv, "experiment$treat == 1 & experiment$st_louis_city == 0")
```

Condition | Overall | Attempted | Canvassed
--------| ----------| ----------| ----------
Control, City | `r control.more.1k$overall.mean` (`r control.more.1k$overall.se`) | `r control.more.1k$attempted.mean` (`r control.more.1k$attempted.se`) | `r control.more.1k$canvassed.mean` (`r control.more.1k$canvassed.se`)
Placebo, City | `r placebo.more.1k$overall.mean` (`r placebo.more.1k$overall.se`) | `r placebo.more.1k$attempted.mean` (`r placebo.more.1k$attempted.se`) | `r placebo.more.1k$canvassed.mean` (`r placebo.more.1k$canvassed.se`)
Treatment, City | `r treat.more.1k$overall.mean` (`r treat.more.1k$overall.se`) | `r treat.more.1k$attempted.mean` (`r treat.more.1k$attempted.se`) | `r treat.more.1k$canvassed.mean` (`r treat.more.1k$canvassed.se`)
--------| ----------| ----------| ----------
Control, County | `r control.less.1k$overall.mean` (`r control.less.1k$overall.se`) | `r control.less.1k$attempted.mean` (`r control.less.1k$attempted.se`) | `r control.less.1k$canvassed.mean` (`r control.less.1k$canvassed.se`)
Placebo, County | `r placebo.less.1k$overall.mean` (`r placebo.less.1k$overall.se`) | `r placebo.less.1k$attempted.mean` (`r placebo.less.1k$attempted.se`) | `r placebo.less.1k$canvassed.mean` (`r placebo.less.1k$canvassed.se`)
Treatment, County | `r treat.less.1k$overall.mean` (`r treat.less.1k$overall.se`) | `r treat.less.1k$attempted.mean` (`r treat.less.1k$attempted.se`) | `r treat.less.1k$canvassed.mean` (`r treat.less.1k$canvassed.se`)

Note: Each cell denotes the turnout rate (mean and standard error of the mean) for each condition and by precinct type at each stage in the experiment.

To estimate a complier average causal effect (CACE) pooled across the two types of precincts, we compare the turnout rates among just those voters canvassed in the treatment and placebo conditions. We do this by regressing turnout on an indicator for treatment and an indicator for St Louis City or County. In one model, we also include covariates from the 2014, 2012, 2010, 2008, and 2006 general elections and the 2016 primary election, following the same PAP plan details discussed above. Furthermore, all standard errors are cluster-robust at the household level, which was the unit of treatment assignment.

```{r, include=FALSE, message=FALSE}
mo.gotv <- subset(mo.gotv, canvassed == 1 & treat != -1)
vars <- c("general14", "general12", "general10", "general08",
          "general06", "primary16")
x <- as.matrix(mo.gotv[,vars])

mo.no.covars.gotv <- round(cl(lm(mo.gotv$voted16 ~ mo.gotv$treat + mo.gotv$st_louis_city), mo.gotv$hh_id)[2,], digits = 4) 
mo.no.covars.gotv[c("Estimate", "Std. Error")] <- mo.no.covars.gotv[c("Estimate", "Std. Error")] * 100 #Get in PP

mo.covars.gotv <- round(cl(lm(mo.gotv$voted16 ~ mo.gotv$treat + mo.gotv$st_louis_city + x), mo.gotv$hh_id)[2,], digits = 4) 
mo.covars.gotv[c("Estimate", "Std. Error")] <- mo.covars.gotv[c("Estimate", "Std. Error")] * 100 #Get in PP
```

Without covariates, we estimate a treatment effect of `r mo.no.covars.gotv["Estimate"]` (SE = `r mo.no.covars.gotv["Std. Error"]`). With covariates, we estimate a treatment effect of `r mo.covars.gotv["Estimate"]` (SE = `r mo.covars.gotv["Std. Error"]`).

Nevertheless, we urge caution when interpreting these results given the covariate imbalance discussed above.

# Identification Strategy for Difference-in-Differences

The difference-in-difference studies included five waves of surveys conducted over the final weeks of the campaign, with the final wave on election day.

Our analyses estimated the following equation:
$$
y_{i,t} = \gamma_{t} + \tau w_{i,t} + \alpha_{i} + \mu_{i,t}, t=0,\ldots,4; i=1,\ldots,N,
$$
where $\gamma_{t}$ is an indicator for the time period, $w_{i,t}$ is an indicator for whether individual $i$ was canvassed before $t$ (such that as soon as a voter is canvassed between $t-1$ and $t$, this indicator is set to 1 and then is then always coded as canvassed thereafter), $\alpha_{i}$ is an individual-level fixed effect, $\mu_{i,t}$ are the idiosyncratic errors clustered at the individual level, and $\tau$ is the treatment effect of canvassing that we are estimating. Below, we present placebo tests of the parallel trends assumption and additional robustness tests. The identification strategy of the differences- in-differences designs rests on the fact that we have precise measures of voters' preferences both before and after they were contacted and a large group of voters who happened never to be con- tacted that allow us to estimate how the electorate's opinions were changing over time regardless. Importantly, in these difference-in-differences studies, we observe which voters the partner group actually contacted and are not relying on voter self-reports of campaign contact.

# NC Difference-in-Differences, 2016, President, Senate, Governor, Supreme Court

For this analysis, we conducted 5 waves of a panel survey, with treatment canvasses delivered throughout. The first wave was conducted around September 20 (n=6,202). The second wave was conducted from 21-29 October (n=3,070). The third wave was conducted from 28 October - 1 November (n=2,876). The fourth wave was conducted from 1-7 November (n=3,285). The final wave was conducted from 8-9 November (n=2,857). Canvassing took place from 26 September - 8 November. For every individual, we know the date when Working America attempted them and when they were successfully canvassed. This allows us to compare the change over time in vote choice among those canvassed to those not canvassed using a difference-in-differences analysis. 

## Universe

The experiment included 6,202 unique individuals. 20% identify as African American, 47% are Democrats, 6% are Republicans, and the remainder are not registered with a party. 15% were attempted with a canvass by Working America and, of those, 15% were successfully canvassed.  

## Tests of Trends Assumption

First, we regress the lagged outcome on an indicator for whether or not an individual is ever canvassed. In each table, we regress the lagged dependent variable from the time period before the individual was canvassed and compare those individuals to everyone who was never canvassed. Standard errors are reported in parantheses. Note that we do not report results for those individuals canvassed between t0 and t1 because this would be the difference in means at the baseline of t0 rather than a within-subject change. 

Lagged Presidential DV | Canvassed by t2 | Canvassed by t3 | Canvassed by t4
--------| ----------| ----------| ----------
Canvassed | 0.19 (0.17) | -0.01 (0.23) | 0.15 (0.13)
t1 | 0.08 (0.01) | 0.08 (0.01) | 0.08 (0.01) 
t2 | n/a | 0.07 (0.01) | 0.07 (0.01)
t3 | n/a | n/a | 0.07 (0.01)
Constant | -0.03 (0.01) | -0.03 (0.01) | -0.03 (0.01)
N obs | 9072 | 11847 | 15048
N groups | 6095 | 6091 | 6095

Lagged Senate DV | Canvassed by t2 | Canvassed by t3 | Canvassed by t4
--------| ----------| ----------| ----------
Canvassed | 0.26 (0.15) | -0.26 (0.20) | 0.04 (0.15)
t1 | 0.04 (0.01) | 0.04 (0.01) | 0.04 (0.01) 
t2 | n/a | 0.09 (0.01) | 0.09 (0.01)
t3 | n/a | n/a | 0.08 (0.01)
Constant | -0.002 (0.01) | -0.002 (0.01) | -0.002 (0.01)
N obs | 9072 | 11847 | 15048
N groups | 6095 | 6091 | 6095

Lagged Governor DV | Canvassed by t2 | Canvassed by t3 | Canvassed by t4
--------| ----------| ----------| ----------
Canvassed | 0.16 (0.18) | -0.05 (0.20) | 0.21 (0.12)
t1 | -0.03 (0.01) | -0.03 (0.01) | -0.03 (0.01) 
t2 | n/a | -0.04 (0.01) | -0.04 (0.01)
t3 | n/a | n/a | -0.04 (0.01)
Constant | -0.02 (0.01) | -0.02 (0.01) | -0.02 (0.01)
N obs | 9072 | 11847 | 15048
N groups | 6095 | 6091 | 6095

Lagged Supreme Court DV | Canvassed by t2 | Canvassed by t3 | Canvassed by t4
--------| ----------| ----------| ----------
Canvassed | 0.33 (0.21) | -0.10 (0.15) | -0.03 (0.20)
t1 | 0.20 (0.02) | 0.19 (0.02) | 0.19 (0.02) 
t2 | n/a | 0.34 (0.03) | 0.34 (0.02)
t3 | n/a | n/a | 0.45 (0.03)
Constant | -0.01 (0.01) | -0.01 (0.01) | -0.01 (0.01)
N obs | 9072 | 11847 | 15048
N groups | 6095 | 6091 | 6095

These four tables suggest that, across the various outcome measures, parallel trends appears to hold. Below, we graphically present these results. 

\includepdf[pages={1},scale=0.75]{"nc_parallel_trends"}

## Description of Treatment

See the description of treatment in the "NC Experiment, 2016, President, Senate, Governor, Supreme Court" section. The same treatment was used. 

## Outcome Measures

See the description of outcome measures in the "NC Experiment, 2016, President, Senate, Governor, Supreme Court" section. The same outcomes were used. 

## Results

Below, we present results where we compare the effect of being canvassed on our four outcome measures. In the first column, we compare those canvassed to all voters who took the baseline survey. In the second column, we compare those canvassed only to those attempted with a canvass by Working America. In all cases, we include time period and individual fixed effects and cluster standard errors at the individual level. Cluster-robust standard errors are reported in parantheses. 

Outcome | Everyone | Among Those Attempted
--------| ----------| ----------
President | -0.02 (0.03) | 0.01 (0.03)
Senate | 0.05 (0.06) | 0.07 (0.06)
Governor | 0.07 (0.04) |0.06 (0.04)
Supreme Court | 0.14 (0.11) | 0.12 (0.11)
N obs | 18,290 | 3,894
N groups | 6,202 | 904

# OH Difference-in-Differences, 2016, President and Senate

For this analysis, we conducted 5 waves of a panel survey, with treatment canvasses delivered throughout. The first wave was conducted from 7-19 October (n=3,545). The second wave was conducted from 20-29 October (n=1,823). The third wave was conducted from 25 October - 1 November (n=1,621). The fourth wave was conducted from 1-7 November (n=1,649). The final wave was conducted from 8-9 November (n=1,328). Canvassing took place from 7 October - 7 November. For every individual, we know the date when Working America attempted them and when they were successfully canvassed. This allows us to compare the change over time in vote choice among those canvassed to those not canvassed using a difference-in-differences analysis. 

## Universe

The experiment included 3,545 unique individuals. 8% identify as African American, 35% are Democrats, 19% are Republicans, and the remainder are not registered with a party. 11% were attempted with a canvass by Working America and, of those, 41% were successfully canvassed.  


## Tests of Trends Assumption

First, we regress the lagged outcome on an indicator for whether or not an individual is ever canvassed. In each table, we regress the lagged dependent variable from the time period before the individual was canvassed and compare those individuals to everyone was is never canvassed. Standard errors are reported in parantheses. Note that we do not report results for those individuals canvassed between t0 and t1 because this would be the difference in means at the baseline of t0 rather than a within-subject change. Furthermore, only 3 individuals were canvassed between t3 and t4, hence the large standard errors for that column. 

Lagged Presidential DV | Canvassed by t2 | Canvassed by t3 | Canvassed by t4
--------| ----------| ----------| ----------
Canvassed | 0.12 (0.16) | -0.01 (0.13) | -0.17 (0.72)
t1 | -0.03 (0.01) | -0.03 (0.01) | -0.03 (0.01) 
t2 | n/a | -0.03 (0.01) | -0.03 (0.01)
t3 | n/a | n/a | -0.04 (0.01)
Constant | 0.01 (0.02) | 0.01 (0.02) | 0.02 (0.02)
N obs | 5120 | 6722 | 8065
N groups | 3413 | 3449 | 3387

Lagged Senate DV | Canvassed by t2 | Canvassed by t3 | Canvassed by t4
--------| ----------| ----------| ----------
Canvassed | -0.19 (0.16) | 0.01 (0.11) | -0.16 (0.64)
t1 | 0.004 (0.01) | 0.01 (0.01) | 0.004 (0.01) 
t2 | n/a | 0.01 (0.02) | 0.01 (0.02)
t3 | n/a | n/a | -0.02 (0.02)
Constant | -0.02 (0.02) | -0.02 (0.02) | -0.02 (0.02)
N obs | 5120 | 6722 | 8065
N groups | 3413 | 3449 | 3387

These two tables suggest that, across the various outcome measures, parallel trends appears to hold. Below, we graphically present these results. Recall that only 3 individuals were canvassed between t3 and t4 and that none of them completed the t3 survey.

\includepdf[pages={1},scale=0.75]{"oh_parallel_trends"}

## Description of Treatment

\includepdf[pages={1-},scale=0.75]{"scripts/OH Diff_2016"}

## Outcome Measures

President:

1. Thinking about the current presidential election, if the presidential election were being held today between Democrat Hillary Clinton, Republican Donald Trump, Libertarian Gary Johnson and Green Party candidate Jill Stein, who would you vote for?
2. Do you have a favorable or unfavorable opinion of Hillary Clinton?
3. Do you have a favorable or unfavorable opinion of Donald Trump?

Senate:

1. Do you approve or disapprove of the way Rob Portman is handling his job as senator?
2. Do you have a favorable or unfavorable opinion of Ted Strickland?
3. Ohio also has a Senate election this fall between current Senator Republican Rob Portman and Democrat Ted Strickland. How do you plan on voting?
4. When it comes to representing Ohio in the U.S. Senate, which candidate do you think is best qualified, Democrat Ted Strickland or Republican Rob Portman?

## Results

Below, we present results where we compare the effect of being canvassed on our two outcome measures. In the first column, we compare those canvassed to all voters who took the baseline survey. In the second column, we compare those canvassed only to those attempted with a canvass by Working America. In all cases, we include time period and individual fixed effects and cluster standard errors at the individual level. Cluster-robust standard errors are reported in parantheses. 

The effect of canvassing on president is statistically significant, but substantively small. Given the greater likelihood of bias under the difference-in-differences assumptions than under those of the randomized experiments, we urge caution when interpreting these results. 

Outcome | Everyone | Among Those Attempted
--------| ----------| ----------
President | 0.055 (0.025) | 0.064 (0.029)
Senate | -0.0165 (0.041) | 0.008 (0.046)
N obs | 9906 | 1665
N groups | 3545 | 389  





